1. Field of the Invention
The present invention relates generally to video processing, and in particular to methods and mechanisms for generating priorities for pixel fetch requests within a digital system.
2. Description of the Related Art
Displays are incorporated into a wide variety of systems and devices such as smart phones, tablets, netbooks, notebook computers, and other devices. These systems and devices include functionality for generating images and data, including video information, which are subsequently output to a display device. Such devices typically include video graphics circuitry to process digital images and video information for subsequent display.
In digital imaging, the smallest item of information in an image is called a “picture element,” or more generally referred to as a “pixel.” For convenience, pixels are typically arranged in a regular two-dimensional grid. By using such an arrangement, many common operations can be implemented by uniformly applying the same operation to each pixel independently. Since each pixel is an elemental part of a digital image, a greater number of pixels can provide a more accurate representation of the digital image. To represent a specific color on an electronic display, each pixel may have three values, one each for the amounts of red, green, and blue present in the desired color. Some formats for electronic displays may also include a fourth value, called alpha, which represents the transparency of the pixel. This format is commonly referred to as ARGB or RGBA. Another format for representing pixel color is YCbCr, where Y corresponds to the luma, or brightness, of a pixel and Cb and Cr correspond to two color-difference chrominance components, representing the blue-difference (Cb) and red-difference (Cr).
Most images and video information displayed on display devices such as LCD screens are interpreted as a succession of image frames, or frames for short. While generally a frame is one of the many still images that make up a complete moving picture or video stream, a frame can also be interpreted more broadly as simply a still image displayed on a digital (discrete, or progressive scan) display. A frame typically consists of a specified number of pixels according to the resolution of the image/video frame. Most graphics systems use frame buffers to store the pixels for image and video frame information. The term “frame buffer” therefore often denotes the actual memory used to hold picture/video frames. The information in a frame buffer typically consists of color values for every pixel to be displayed on the screen. Color values are commonly stored in 1-bit monochrome, 4-bit palletized, 8-bit palletized, 16-bit high color and 24-bit true color formats. The total amount of the memory required for frame buffers to store image/video information depends on the resolution of the output signal, and on the color depth and palette size.
The frame buffers can be situated in memory elements dedicated to store image and video information, or they can be situated in the system memory. Consequently, system memory may be used to store a set of pixel data that defines an image and/or video stream for display on a display device. Typically, applications running in such a system can write the pixel data into the system memory, from where the pixel data may be obtained to eventually generate a set of image/video signals for generating the image on the display device. In such systems, fetching the frames (pixel information) from system memory may place high demands on the system, as other devices may also be competing for memory access. As consequence, a high bandwidth may be required from memory in order to keep up with the requests for data. In addition, as each system memory access requires a certain amount of processing power, requests for high volume pixel data may eventually result in premature battery depletion in battery-operated devices, such as mobile phones, tablets, and notebook computers.
The design of a smartphone or tablet may include user interface layers, cameras, and video sources such as media players. Each of these sources may utilize video data stored in memory. A corresponding display controller may include multiple internal pixel-processing pipelines for these sources. Each of these pixel-processing pipelines may make requests to memory for the corresponding source frames.
In addition to the pixel-processing pipelines, a typical device also has numerous other non-display related functional units that need access to memory (e.g. processors, peripherals, etc.). For example, a processor accesses memory to read instructions for execution, to read and write data during execution of the instructions, etc. A network device reads and writes packet data to/from memory. A mass storage device writes stored data being transferred to memory, or reads memory data being transferred to the mass storage device
Each memory request sent from one of the multiple sources includes both overhead processing and information retrieval processing. A large number of requests from separate sources of the device may create a bottleneck in the memory subsystem. The repeated overhead processing may reduce the subsystem performance.
With numerous devices potentially accessing memory, a mechanism for selecting among requests, ordering requests from different requestors, etc. is needed. The mechanism needs to balance performance requirements of the requestors (which differ, depending on the type of requester) as well as providing good memory performance (e.g. grouping operations to the same page of memory to improve memory bandwidth utilization and reduce average power consumption, etc.).
Some devices are categorized as real-time devices. These devices are characterized by a need to receive data at a certain rate in real time, or erroneous operation may occur. For example, video data needs to be provided within the frame rate of the video, or visual artifacts may occur on the display. Similarly, audio devices are real time. If the audio data is not available at the audio rate, skips in the audio playback may occur. Other devices are non-real time, such as processors. Non-real time devices can perform better if data is provided more rapidly, but will not have erroneous operation if data is not provided as rapidly.
One mechanism that can be used to balance the requirements of real time and non-real time device is quality of service (QoS). The real time devices can be provided with several levels of QoS, with increasing levels of priority. As the need for data becomes more critical to prevent erroneous operation, the device can issue memory operations with higher levels of QoS. The memory controller can respond more rapidly to this higher QoS requests, preventing the erroneous operation that might otherwise occur.
There are costs for issuing the higher QoS requests, at the system level. The memory controller may bypass other requests that might be more efficiently performed together (e.g., requests that are to an already-open page in the memory). Accordingly, overall system performance can suffer if the QoS levels of real time requests are increased too frequently. Because erroneous operation occurs for real time devices if their data needs are not met, the determination of which QoS level to use for a given memory operation may be made conservatively (i.e., assuming a worst case scenario in terms of memory load from other requestors in the system). However, while such determinations can ensure the correct operation of the real time devices, the increase in QoS levels can occur more frequently than necessary if the worst case scenario is not in effect, reducing memory bandwidth utilization and increasing power consumption in the memory unnecessarily.